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Abstract 

Within the Standard model with the 4th generation quarks b' and t' we have analyzed CP- 
violating flavor changing neutral current processes t — > cX, b' — > sX, b' — > bX, t' — > cX, 
and t' — > t X, with X = H, Z, 7, g, by constructing and employing global, unique fit for the 
4th generation mass mixing matrix CKM4 at 300 < m t i < 700 GeV. All quantities appearing 
in the CKM4 were subject to our fitting procedure. We have found that our fit produces the 
following CP partial rate asymmetry dominance: acp(b' — > s (H, Z; 7, g)) ~ (94, 62; 47, 41) 
%, at m t i ~ 300; 350 GeV, respectively. From the experimental point of view the best 
decay mode, out of the above four, is certainly b' — > 37, due to the presence of the high 
energy single photon in the final state. We have also obtained relatively large asymmetry 
acp{t — > eg) ~ (8 — 18) % for t' running in the loops. There are fair chances that the 4th 
generation quarks will be discovered at LHC and that some of their decay rates shall be 
measured. If b' and t' exist at energies we assumed, with well executed tagging, large acp 
could be found too. 
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1 Introduction 



In this paper the main idea is to find possibly large genuine CP violation (CPV) effects in the 
decays of the fourth generation of quarks, arising from the one-loop flavor changing neutral 
currents (FCNC), by using unique fitting procedure. 

A fourth generation of quarks and leptons, which we refer as (£', b', £', v'\ in our opinion 
is one of the most conservative guesses one could make as to what new physics lies ahead. 
Since the 4th generation of flavors is neither predicted nor disallowed by the Standard Model 
(SM3) we should keep an open mind regarding its existence. 

Since possible existence of the 4th generation provides a number of desirable features, for 
a review see [T] and for a more updated review see [2], let us first remind the reader of those. 

* Fourth family is consistent with electroweak (EW) precision tests [3], because, if the 
4th generation of fermions satisfy the following constraint for quarks [4] 

mv -m v ~(l + ^ In n ^? e v ) X 55 ° eV ' (L1) 

and the related 4th lepton generation mass difference my —m u / ~ 60 GeV [4], the electroweak 
oblique parameters [5] are not extending the experimentally allowed parameter space. 

* The 4th family of fermions is consistent with SU(5) gauge coupling, i.e. it can be unified 
without supersymmetry, and, because of (11. ip . the 4th generation softens the current Higgs 
bounds [6]. 

* With respect to quark-neutrino physics, if the 4th generation were discovered, it may 
change our prediction for the K + — > 7i + uu decay [7J. There can be also implications on other 
penguin-induced decays, like b — > and b — > s<f), see [8] and [9], respectively. 

* A heavy 4th family could naturally play a role in the dynamical breaking of EW sym- 
metry pill [IT]. 

* If the unitarity of SM3 CKM matrix V^ x3 = Vckm3 is slightly broken, new informations 
from top-quark production at Tevatron still leaves open possibility that |V#,| is nontrivialy 
smaller than one [T2] . 

* In addition, a new generation might also cure some flavor physics problems too [TOf [13]. 

* Finally, the 4th family might solve baryogenesis related problems, by visible increase of 
the measure of CP violation and the strength of the phase transition. Namely, the question 
of the large CP-violation in the SM3 extended to the 4th generation of fermions (SM4) 
was recently raised in [14], with respect of the insufficient CP asymmetry produced in the 
Standard model with three generation for generating the baryon asymmetry in the universe. 

There is a known quantity, the Jarlskog invariant, which measures the CP violation in 
the model. For SM3, it is defined as [T7] 

J = {m 2 t - ml){m 2 t - m 2 c ) (m 2 c - m 2 u ) (m 2 b - m 2 d ){m 2 b - m 2 s ){m 2 s - m 2 d )A, (1.2) 

where A is the twice of the area of any of six unitary triangles in SM3 and it is of order of 
O(10" 5 ). 

The fourth generation is added to the three known SU (2) / doublets as 
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So in the case of SM4, the above relation (11.21) in the d — s degeneracy limit generalizes to 

M 

4a4 = {ml ~ m 2 c ){m 2 tl - m 2 t ){m 2 t - m 2 c ){m 2 b , - m 2 ){m 2 b , - m 2 b ){m 2 b - m 2 )A b 2 s M , (1.4) 

which, due to the large my , m t t masses and somewhat larger area of the b — > s quadrangle 
A234 corresponding to the SM4 unitarity relation V 4x4 V 4x4: = 1, can be up to 15 orders of 
magnitude larger than the Jarlskog invariant in the SM3 [14] . However, this enhancement 
of CPV in the model with 4th fermion generation cannot, by itself, solve the problem of the 
baryogenesis, since just adding the fermions reduces the electroweak phase transition if there 
are not some additional theories involved like supersymmetry [15J , or the theory with at least 
two Higgs doublets [16]. 

We start our analysis by performing a fit of SM4 CKM mass matrix. To obtain valid 
CPV results it is also crucial to follow results of the general fit of the electroweak precision 
data, since, in order to be able to calculate the real value of ^2345 ^ i s necessary to make a 
global fit of a complete matrix. Therefore, on top of many different processes used in 
the fit, we have also taken into account the EW constraints on the CKM mixing between the 
3rd and the 4th quark family. 

We than compute flavor changing neutral current (FCNC) processes of the fourth gen- 
eration quarks, like b' — > s(H, Z,j, g), etc., and analyze the most important consequence: 
large CP- violation in such decays. The rare top decays t —>■ c{H.,Z^,g\ involving the 4th 
generation quarks running in the loops are considered too. 

The paper is organized as follows. In Section 2. we introduce the quark mixing matrix with 
4th generation and construct the fitting procedure, the fit itself, and present the corresponding 
results, respectively. Section 3. contains computations of the FCNC processes involving 4th 
family, while the CP violation effects due to the 4th generation are discussed in Section 4. 
Lastly, Section 5. is devoted to discussions and a conclusion. 



2 CKM matrix for the fourth generation 

The fourth generation 4x4 quark mixing matrix is given by 



Va 



4x4 



Vr 



CKM4 



/ V u d V us V ub V u y \ 

V c d V cs V c b V c b' 

Vtd V ts V t b Vw 

\V t 'd Vfs Vfb Vfb> ) 



(2.1) 



The parameterization of such a matrix can be done in many possible ways. We have chosen 
to use the standard CKM3 Wolfenstein parameterization [TB] of the 3x3 matrix, up to 0(A 5 ) 



Vr 



CKM3 



V u d V us V u b 
V c d V cs V c b 

Vtd V ts Vtb 



( 



2 



A 4 



A A\ 3 (p-ir})\ 
(-1 + A% (1 - 2{p + i V )fj 1 - f - £(1 + AA 2 ) AX 2 

\ AA 3 (l-(p + ^)(l-f)) AA 2 (-l + f (l-2(p + M7))) l-A% J 



(2.2) 
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and to multiply it by the mixing matrices of the first, the second and the third generation 
with the fourth generation, i? 14 , i? 2 4, -R34, respectively, in a following way [T9] : 



VcKM4 — -^34 • -^24 • Rl4 ■ VcKM3 , 



(2-3) 



where 



1 1 \ 
10 

c u s u 
\0 -s u c u J 



(2.4) 



/l \ 

c v s v e-^ 2 
10 



(2.5) 



\0-s„e i<fc c v J 



( c 



s w e-^ 3 \ 

1 
1 








C w J 



(2.6) 



The values for the parameters A, A, p and 77 are taken to be in the range given by the global 
fit in SM3, [20]. The fourth generation parameters, c U}VjW = cos9 UtVtW and s UjVjW = sin9 UtV!W , 
and the two new phases 2i3 (s^^ = sin0 2i3 ) are the new parameters which need to be 
fitted. The label is reserved for the standard CKM3 phase appearing in (12.21) . In this 
parameterization, all matrix elements will now depend on the new parameters, for example, 
the matrix element V u d will have the form 



In order to estimate CPV phenomena with the 4th generation quarks, first we have to 
determine elements of the new 4x4 quark mixing matrix Vckm4, (E3J), which essentially 
means to do a fitting of the 4th generation parameters 



We shall perform the fit of these parameters by analyzing K° — K°, D° — D° and B® s — B® s 
mixings, and estimating the decays K + — > tt + uV and B —>■ X s 7. The fit to the new measure- 
ment of sin 2f3^x is added too. Moreover, in this analysis we are following strict requirement 
of the unitarity condition of the new matrix (12.31) at the expense of slight unitarity breaking of 
the CKM3 matrix. This, together with the independently measured CKM3 matrix elements 
[20] . will give us additional constraints on the parameters of the V^xa quark mixing matrix 



2.1 Definition of our fitting procedure 

The fit is performed by the CERN fortran code called Minuit [21]. It minimizes the multi- 
parameter function which is defined as a sum of various x 2 ' s between the fitted expression 




1 S U j S v , S w , S^2, S(^ 3 . 



(2.7) 
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and the data: 

v 2/ n ,x _ V- (th(a)i - expjf 

where th(a)i defines the 4th generation model parameter dependent predictions of a given 
constraint i, and exp represents the measured values. At hi is the uncertainty of prediction 
thi and Aexpi is the uncertainty of the individual measurement expi. a is the vector of free 
parameters being fitted, in our case a = (s u ,s v ,s w , s^ 2 , s^ 3 ). The x 2 w iU i n addition depend 
on the masses of the b' and t' quarks. The fit is performed by varying my and %, in such 
a way that the constraint from the electroweak precision measurements is fulfilled, assuming 
m H = 115 GeV fTLTj) : 

m y = mtl - 55 GeV . (2.9) 

Much larger mass splitting would require more tuning in the canceling contributions to the 
EW T-parameter. 

A remark is in order: due to the complexity of the expressions, the uncertainties of the 
theoretical predictions are not taken into the fit. 

2.2 Fitting different measured processes in the model with four 
generations 

2. a. Check of the unitarity of the CKM3 matrix in SM3 

First, we use the unitarity bound on the CKM3 parameters coming from the independent 
measurements. These are: 

= 0.9999 ± 0.0011 (1st row) , 
1.136 ±0.125 (2nd row), 
1.002 ±0.005 (1st column), 
1.134 ±0.125 (2nd column). (2.10) 
Also, the six CKM3 matrix elements are measured independently, 

\V ud \ = 0.97418 ±0.00027, 
\V US \ = 0.2255 ±0.0019, 
\V cd \ = 0.230 ±0.011, 
\V CS \ = 1.04 ±0.06, 
\V cb \ = (41.2 ± 1.1) x 10~ 3 , 

\V ub \ = (3.93 ± 0.36) x 10~ 3 , (2.11) 
which give us additional six constraints on the unknown parameters. 



\v ud 


r± 


\v us 


r± 


\v ub \ 2 


\v cd \ 


2 ± 


\Vcs\ 


2 ± 


\Vcb\ 2 


\v ud 


i 2 ± 


\v cd 


i 2 ± 


\v td \ 2 


\v us \ 


2 ± 


\v cs \ 


2 ± 


\Vts\ 2 
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2.b. K° - K° mixing 

We modify expressions for the AM^, €k and e'/e ratio in the model with the fourth 
generation in order to limit the elements Vfd and Vf s . The experimental values are: 

AM K = (3.483 ± 0.006) • 10~ 15 , (2.12) 

e K = \e\ = (2.229 ± 0.012) • 10~ 3 , (2.13) 

Re(e'/e) = (1.63 ± 0.26) • 10~ 3 . (2.14) 

2.c. D° — D° mixing 

An expression for the xp is used in order to determine V c y and V u b> . From the experiment 
we have: 

x D = 0.776 ±0.008. (2.15) 

2-d. B° d s - B d s mixings 

We need the x Bd and xb s mixing parameters in order to bound V t >d , Vfh , and V t ' s , V t >b , 
respectively. The measured values are: 

x Bi = 0.776 ±0.008, (2.16) 
x Bs = 26.1 ±0.5. (2.17) 

2.e. K + — > 7i + uu process 

From the branching ratio for K + — > n + uu we confine Vfd and Vf s - Recent experiments 
give: 

BR(K + -> tt+z/z/) = (1.5 ± 1.3) x 10" 10 . (2.18) 

2.f. B — > X s 7 process 

To find Vt'b we make also use of the branching ratio for B — > X s ^ and employ 

BR(B -> X s7 ) = (3.55 ± 0.26) x 10" 4 . (2.19) 

2.g. sin 2/5 from 5 -> J/^iT 

In the SM3, the best measurement of the sin 2(3 comes from B — > J/ipK decay, giving 

» = ("©■ (2 ' 20) 

In the model with the fourth generation, the sin 2j3 can get modified with a new phase, i.e. 
sin 2(3 —>■ sin(2/3 — 26). So, we need to manipulate the expression for 9 in order to determine 
Vfb and Vfd from the experimental data, 

sin 2(3 = 0.681 ±0.025. (2.21) 

It is important to note that in all above processes the mass m t > appears explicitely in the fit, 
except for D° — D° mixing, which depends on the my mass (12. 15p . 
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The analytic formulae for the processes are taken from various papers: for kaon mixing 
and decays from [22]; for D° — D° mixing from [23J; for B% s — B® s mixings from [21]; for 
B — > Xg'j from |25j . 

Various loop-induced processes depend on different Inami-Lim functions [26]. The inclu- 
sion of the 4th generation quarks in the loops brings additional Inami-Lim functions depend- 
ing now on mt> 7 b'/Mw arid the products of the new CKM4 matrix elements X b /, X b t f, Xf (and 
similarly for b'), where 

K m = V k * l V km , (2.22) 

see for example the analysis in [27] . 

All QCD lattice parameters above are taken from the averages in [21]; see also [29J. 

We comment here that as was pointed out in [30j, there is tension between the data on 
Bj— mixing and sin (2/3) and the theoretical predictions for SM4, based on the lattice QCD 
calculations of ref. [31J. This may signal new physics, such as SM4. Therefore, it is very 
important to obtain definite results for the parameters calculated in lattice QCD. 

In addition, we take into account the findings of two recent studies [321 [33] on the 4th 
generation mixing with the standard three quark families. In the first paper [32J , the authors 
perform similar fit like our, by using experimental constraints coming from the measured 
CKM3 matrix elements and FCNC processes (K—,D—,Bd~,B s — mixings and the decay 
b — > 57) and asumming the unitarity of the new matrix. As it can be seen from above, 
we have extended the fit adding more FCNC constraints, but our results closely follow the 
findings of [32], in a sense that the large mixing between 3rd and 4th generation is allowed 
for some range of the five-dimensional fitting space a. However, the analysis of second paper 
[3H] have shown that such a large mixing between third and the fourth generation, larger 
than the Cabibbo mixing of the first two families, is excluded by the electroweak precision 
data. Therefore, in addition, we apply the EW precision data constraint from [33], which 
implies that maximum of sin #34 = sin 9 U must be in the following range 

, . „ , f 0.35 ±0.001, for m v = 300 GeV , ooN 
max (sintU = < _ ., . . _ in , „ 2.23 

v ; \ 0.11 ±0.10, for m t t = 1000 GeV, v ; 

(for other values and for more explanations, see Table 3 in [33]). Here, the lower bound for 
large m t > masses is enlarged, due to the unreliable perturbation theory applied for the EW 
fits at such large energies (see discussion in [33]). 

Applying all the constraints discussed above, we obtain the results presented in the next 
subsection. 



2.3 The results of the fitting procedure 

Here are the results for the fitted values of the vector a = (s u , s v , s w , S0 2 , s^) depending on 
the 4th generation quark masses. Since we have just one place where the my mass enters, 
(2.c.) from above, the quark mass dependence comes mainly from the m t i. 
The experimental constraints on the m t i and my masses are [20, 34J: 

my > (46 - 199) GeV, (2.24) 

m t , > 256 GeV. (2.25) 
Therefore, we scan the m t i in the range of 

300 GeV < 77V < 1000 GeV , (2.26) 
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mt'(GeV) 


sin 9 U 


Yn;„/d.O. f 


300 


0.25 ± 0.04 


0.85 


350 


0.13 ± 0.03 


0.98 


400 


0.10 ± 0.02 


0.84 


450 


0.10 ± 0.04 


0.79 


500 


0.10 ± 0.04 


0.80 


600 


0.11 ±0.03 


0.93 


700 


0.11 ±0.02 


1.17 


800 


0.11 ±0.02 


1.45 


900 


0.11 ±0.02 


1.76 


1000 


0.11 ±0.02 


2.07 



Table 1: Results of our fit on the mixing between the third and the fourth generation obtained 
including the EW constraints from [53] . 



m t /(GeV) 


300 


400 


500 


600 


700 


sin 6 U 


0.25 ±0.004 


0.10 ±0.02 


0.10 ±0.004 


0.11 ±0.03 


0.11 ±0.02 


sin 9 V 


0.010 ±0.003 


0.029 ±0.001 


0.034 ±0.001 


0.033 ±0.008 


0.031 ±0.005 


sin 6 W 


0.002 ±0.001 


0.016 ±0.002 


0.015 ±0.001 


0.014 ±0.001 


0.012 ±0.001 


sin 02 


-0.4 ±0.4 


0.97 ±0.01 


0.947 ±0.002 


0.91 ±0.02 


0.89 ±0.04 


sin 03 


0.2 ±0.3 


0.99 ±0.02 


0.987 ±0.001 


0.96 ±0.03 


0.95 ±0.03 



Table 2: Final results for the 4th generation parameters obtained with the acceptable quality 
fit. 

and take care about the EW precision data limit on my and m t i mass difference, eq. (12. 9p . 

It is important to note that in models with the light Higgs, there is an unitary bound on 
the masses of the fourth generation quarks which amounts to myy < 550 GeV. If the Higgs 
boson is heavy (ran > 500 GeV), the above perturbative limit does not hold, and the masses 
of the fourth family can be larger [2J 133] . 

The quality of the fit is given by the minimal x 2 /d.o.f ', where d.o.f is the number of the 
constraints minus the number of the fitted parameters. The best fit is when Xmm/ d-o.f ~ 1. 
For the numbers given below, d.o.f = 13. 



m^GeV) 


300 


400 


500 


600 


700 


\V*\ 




0.964 ±0.010 


0.993 ± 0.003 


0.993 ± 0.001 


0.992 ± 0.003 


0.992 ± 0.003 


\V W 




0.258 ±0.037 


0.107 ±0.022 


0.106 ±0.004 


0.115 ±0.025 


0.118 ±0.028 


\v Vh 




0.259 ±0.037 


0.108 ±0.022 


0.106 ±0.004 


0.115 ±0.025 


0.118 ±0.028 


\Vt'b' 




0.965 ±0.010 


0.993 ±0.003 


0.994 ±0.001 


0.992 ±0.003 


0.992 ± 0.003 


\Ve d 




0.002 ±0.001 


0.009 ±0.002 


0.008 ±0.001 


0.006 ±0.001 


0.006 ±0.001 


\Vtf 9 




0.005 ±0.005 


0.031 ±0.001 


0.034 ±0.007 


0.034 ±0.009 


0.031 ±0.005 


\Vub' 




0.002 ±0.001 


0.016 ±0.002 


0.016 ±0.004 


0.014 ±0.001 


0.013 ±0.002 


\V c b> 




0.010 ±0.003 


0.030 ±0.001 


0.033 ±0.002 


0.034 ±0.009 


0.031 ±0.005 



Table 3: Predictions for the selected Vqkm matrix elements using the best fit from Table 2. 
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The following results of the fitting procedure are of a special importance: 

i) m t > ~ [300 — 600] GeV region is preferred by the x 2 scan, i.e. Xmin/d-o.f ~ 1. The larger 
m t r masses do not produce a good fit, as one can see from the Table 1. 

ii) the best fits with xLm/ d.o.f ~ 1 for m t i > 600 GeV give too large s u = sin9 u mixing 
angle which is excluded by the EW precision data. On the other hand, the allowed values 
for s u are obtained with the bad fit, with Xmm/d.o.f. > 1, see Table 1. 

iii) in addition, we test the predictions for all quantities entering the fit using the new fitted 
parameters, in a way that we look for the 'pull'(= (data central value - predicted value) /(data 
error) ) of the data. So, although the fit for m t > = 700 GeV has Xmm/ d.o.f slightly larger 
than 1, we have decided to keep this fit, since the predictions with this mass of m t > nicely 
match with the data. 

Having in mind all the facts above, we conclude that our best fits are obtained for 300 < 
m t i < 700, with the fitted parameters given in Table 2., while the selected Vckm matrix 
elements are presented in Table 3. 

The final results at 95% confidence level of the complete 4x4 fitted matrices are given 
below: 



V C KM4(m t/ = 300 GeV) = 



VcKM4(mf = 400 GeV) 



V C KM4(m f = 500 GeV) 



/ 0.9742 0.2257 0.0035e~ 68 - 9Oi 0.0018e~ 12 - 4Oi \ 

-0.2255 0.9732 0.0414 0.0102e 29 - 8 °* 

0.0086e- 241Oi -0.0416e°- 7Oi 0.9649 0.2589 
-0.0019e 18 9Oi 0.0052e 69 - 3Oi -0.2591 0.9658 



(2.27) 



/ 0.9740 0.2256 0.0036e~ 68 - 9Oi 0.0164 e - 874Oi \ 

-0.2259 0.9728 0.0414 0.0290e~ 761Oi 

0.0092e- 27 - 7Oi -0.0414 0.9932 0.1079 

\-0.0091e 89 - 9Oi 0.0310e" 94 - 6Oi -0.1082 0.9935 / 



(2.28) 



/ 0.9740 0.2256 0.0035e~ 68 9Oi 0.0160e~ 

-0.2259 0.9726 0.0414 0.0329e" 

0.0083e" 271Oi -0.0416e 6 ° Oi 0.9934 0.1059 

\-0.0080e 83 2Oi 0.0344e- 100 - 4Oi -0.1062e a7Oi 0.9937 



■81.1°i \ 
•71.8°j 



(2.29) 



VcKM4(m t , = 600 GeV) 



/ 0.9741 0.2256 0.0035e- 68 9Oi 0.0140e~ 75 - 4Oi \ 

-0.2258 0.9726 0.0414 0.0339e- 66 ° Oi 

0.0089e~ 26 - 2Oi -0.0423e 63Oi 0.9924 0.1149 

\-0.0058e 77 - 9Oi 0.0343e- 105 - 9Oi -0.1155e a7Oi 0.9926 



(2.30) 
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VcKMt{mt> = 700GeV) = 



/ 0.9741 0.2256 0.0035e- 689 °* 0.0130e- 72 - 9 °* \ 

-0.2258 0.9727 0.0414 0.0309e- 62 - 9 °< 

0.0088e" 26 - 2Oi -0.0423e 5 - 8 °* 0.9920 0.1179 

\-0.0056e 74 7Oi 0.0309e- 1081Oi -0.1185e°' 6Oi 0.9924 / 



(2.31) 

Note, that the fitted parameters show small 4th generation mass dependence in the prefer- 
able range of m t >, excluding the fitted 4x4 matrix at m t i = 300 GeV, (12.271) . 

From the above matrices we can see that the fit exhibits constraint \V t b\ > 0.96, which 
is much stronger than the limit \V t b\ > 0.74 following from the single top quark production 
cross section measurement [20] . 

Comparing our results with those existing in the literature [32], [36] 157] . we can deduce 
that our fit, under the conditions specified in Sec. 2.2, excludes large mixing between 4th 
and the first three generations. Our matrix elements \V u b>\, |V^/|, \Vw\ from Table 3. are 
significantly smaller (up to six times for \V t y\) with respect to the same elements obtained 
by the conservative bound in [32] and in [38]. This is a direct consequence of the applied EW 
constraint on sin# u , (12.231) . since otherwise, as already mentioned at the end of Sec. 2.2, the 
somewhat larger mixing between the third and the fourth generation, relative to the bound 
from eq. (12.231) . is obtained. In [27], the mixing is bounded to sin 6*34 < 0.14. 

Considering phases of CKM4, in our approach they are strongly depending on the m t i 
mass, oscillating widely, as they do in [32]. In [27], as well as in [32J[3S], the fits are performed 
under the assumption that the phases are free and run between and 1-n. However, in our 
global and unique fit, which generates matrices (12.271 - [2731]) . the phases are also subject of 
the fitting procedure. Therefore, the complex interplay between all fitting parameters can 
significantly influence the final allowed parameter values of the matrix elements. 

Although, the standard CKM3 matrix elements, as a part of Vckm4 were fitted, in our fit 
their values (12.271 - I2.3ip do not contradict the global CKM3 fit from [20] . This is especially 
true for the less constrained elements like Vu and V ts . 

The obtained fourth generation parameter values (12.271 - [273TT) will be used in the calcula- 
tion of the rare decay branching ratios and CP partial rate asymmetry in the next sections. 



3 Rare processes involving the fourth generation 

We analyze FCNC decay processes of the fourth generation quarks, in particular of t' — > 
(c,t)X, and b' — > (s,b)X with (X = H, Z, 7, g), arising from the generic one-loop diagrams 
given in Figs{T] We also study the influence of the 4th generation FCNC model to the 
ordinary top quark rare decays: t — > cX. 

The rare FCNC processes of the above type have been extensively studied in the context 
of various extensions of SM. We base our study on the explicit analytical expressions on 
Q — > q(Z, 7, g) given in [39J; with Q = (t, t', b') and q = (c, (t, c), (6, s)), respectively. Checks 
for Q —> (7(7, g) decays are performed using expressions from [40J. The Q — > qH decays were 
considered in [41] . 

To obtain the branching ratios, the decay amplitudes will be normalized to the widths of 
the decaying quarks. For t-quark decays we have 

BR(i cX) = S^TS^y ' ("> 
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Figure 1: Generic diagrams for FCNC decays of the Jfth generation quarks. X denotes 
possible decays to X = H, Z, 7, g and quarks running in the loops are qu = {u, c, t, t'} and 
q D = {d,s,b,b'}. 



while for t! and b' decays, we will also take into account the CKM4-suppressed tree level 
decays. Therefore, 

HR(f -+ (c, t)X) = - W^X) , (3.2) 

v ^ ' ' J r(t> -> bw) + r(f -+sW)' y 1 

ryoliJ < u\ v\ T(b'^(s,b)X) 

where 6' — * tW 7 * is effective for my < 255 GeV. The tree level decays are given by 

G M 3 x 3 1 

r(Q - q w) = g 7r ^ Q l^l 2 V A ^ ^q) 2 , K/^) 2 ) 

x ((1 - xj/x 2 / + 1/4 (1 + xj/x 2 ,) - 2/x 4 Q ) , (3.4) 

where A(x, y, z) = x 2 + y 2 + z 2 — 2xy — 2xy — 2yz and x 2 = mf/M^- The masses running 
in the loops in FigsH] are taken to be current quark masses, while the external masses are 
considered as pole masses. Practically, this makes no numerical difference in the calculation, 
apart for t — > cX decays with 6-quark running in the loops. There we take fn^rrit) = 2.74 
GeV, jlQ], and, with our set of parameters and m t = 171.2 GeV, we obtain the following SM 
results: 

BR SM (t -> try) = 4.4 • 1CT 14 , BR SM {t -> eg) = 3.8 • 10~ 12 , 

BR SM (t -> cZ) = 1.3 • 1(T 14 , BR 5M (t -> c#) = 7.8 • 1(T 15 , (3.5) 

comparable with the estimates given in [4TJI H2]. With the pole masses running in the loops 
[55] , the results become an order of magnitude larger. The values from eqs. (13.51) have to 
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Figure 2: Branching ratios for rare top decays in the model with 4th generation as a function 
of nit', for b' of the mass my = m t > — 55 GeV running in the loops. X denotes possible decays 
to X = H (dashed line), Z (solid line), ^(dotted line), g (dashed- dotted line). 
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Figure 3: Branching ratios of t' — ► (c, t)X as a function of m t i = my + 55 GeV. X denotes 
possible decays to X = H (dashed line), Z (solid line), ^(dotted line), g (dashed- dotted line). 



be compared with the largely enhanced BRs of t — > cX for 4th generation quarks included 
in the loops, FigfSJ The foreseen sensitivities for t —>■ cX channels at Tevatron and/or LHC 
could be sufficient to see these enhanced rates. 

Throughout of the calculation the mass of the Higgs boson is taken to be mu = 115 GeV. 
Our fit favors fourth generation masses slightly larger than the so-called unitary bound of 
~ 600 GeV. In that case the concept of light Higgs boson and the elementary scalar Higgs 
field is no more appropriate, since the Goldston boson of the electroweak symmetry breaking 
would couple very strongly to the heavy 4th generation quarks [2] [55]. Therefore, the results 
for mt> y b' > 600 GeV have to be taken with precaution. 

In the analysis we have also examined the influence of the PF-boson width to the results. 
The inclusion of the finite width for the PF-boson propagating through the loops, [43], is 
effective only for the t —+ cX decays, enhancing BRs by some 10%. 

Our prediction for BR(t — > cH) given in Fig 121 contrary to [36J, always dominates over 
Z, 7, g modes, in the whole range of t' mass. For t' — > (c, t)X (FigJH]) and b' (s, b)X (FigfJJ 
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Figure 4: Branching ratios of b' — > (s, 6)X as a function of m t i = my + 55 GeV. X denotes 
possible decays to X = H (dashed line), Z (solid line), 7 (dotted line), g (dashed- dotted line). 



decay modes general behavior is more or less the same, except that for our global fit, both, 
the gluon and the Higgs modes dominate over Z and photon modes, apart from the case given 
in FigSb, where H and Z dominate over g and 7 modes, respectively. In [36J dominating 
modes are Z and the decay into gluon, which is due to a large difference between our CKM4 
parameters and the parameters used in [36] . 



4 CP violation 

The CP partial rate asymmetry, for decays discussed above, is defined as 

_ T(Q-+qX)-T(Q^qX) 
CP r(Q - qX) + T(Q^qX)- 1 ' ] 

Since the rates involve at least two amplitudes with different CP-conserving strong phases 
coming from the absorptive parts of the loops, while the CP-violating weak phases are pro- 
vided by the phases in Vckm4, we expect to find CP violation in FCNC decays of 4th genera- 
tion quarks [44]. The inclusion of the finite W-boson width can enhance a CP-asymmetry by 
enhancing the CP-conserving phases, but, since this happens almost equally for T(Q — > qX) 
and T(Q — > qX), the effect appear to be at most of 10% level. Estimated CP asymmetries, 
shown in FigslMZl for FCNC rare decay modes, in general oscillate as a function of t' mass. 
In particular, important modes for CPV effects are b' — > sX decays, as also noted recently in 
[57] . For b' — > s(H, Z) modes we find very interesting maximal CP partial rate asymmetry 
at m t i = 300 GeV, i.e. 94% and 62% respectively. These large numbers occur due to the tW 
loop threshold at my ~ 250 GeV. Other two modes, b' —>■ s(j,g), produce maximal CPV at 
m t ' = 350 GeV in the amount of 47% and 41% for 7 and g, respectively, and they could be 
important too, FigJTK Maximal CP partial rate asymmetry for t —>■ cX modes occur also 
at m t i = 350 GeV for t — ► eg, and it amounts to 18%, FigfSJ For t' — > (c,t)X and 6' — ► bX 
modes acp is very small, always bellow 0.25%, Figs lBTTb . 

At the end, let us discuss some general features of the CP violation within the model with 
the 4th family. 



12 



0.2 - 



0.1 



o 

T 

4i 0.0 1# 



-0.1 



-0.2 



300 



400 500 

m,- (GeV) 



600 



700 



Figure 5: Fourth generation effect on the acp of the rare top decays as a function of m t > , for 
b' of the mass my = m t i — 55 GeV running in the loops. X denotes possible decays to X = 
H(.), Z(.), g(.). 
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Figure 6: CP asymmetries in t' — > (c,t)X decays as a function of m t i = my + 55 GeV. X 
denotes possible decays to X = H(m), Z(a), j(m) 7 g(*)- 



Following the analysis of Ref.[45], we calculate the strengths \Bi\ of CP violation for a 
fourth family in the chiral limit Tn u ,d,c — 0. Definitions of the relevant imaginary products in 
the chiral limit are [45J: 

b x = imv cb v; b v t ,yv; b , , (4.2) 

B 2 = ImV tb V t J b V t ,yV t y , (4.3) 
B 3 = lmV cb V t lV t yV; b , . (4.4) 

In the paper [45] a rigorous upper bound on \Bi\ < 10~ 2 in the model with the 4th family was 
obtained. Calculating these quantities explicity for the values of our CKM4 matrix elements 
(I2.27H2.3T1) . we obtain the strengths of the CP violation of the order 

, . f 5 ■ 10~ 5 for m v = 300 GeV, . , 

I #i,2,3 1 - | 10 _ 4 f or mt , = [400 - 700] GeV. ^ ] 
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Figure 7: CP asymmetries in b' —* (s,b)X decays as a function of mf = my + 55 GeV. X 
denotes possible decays to X = H(m), Z(a), jfm), g(*)- 



The area of the unitary quadrangle A w , with the sides V ub V* b , , V cb V* b , , Vt b V t l, > Vt'bVJy 
describing CPV in the chiral limit, is 



A W = \{\B 1 + B 2 \ + \B 1 + B 3 \ 
and with our fitted parameters amounts to 



I -Bo 



\B*\) 



2A bb > 



10" 5 for m t > = 300 GeV, 
4 • 10~ 4 for nv = [400 - 700] GeV. 



(4.6) 



(4.7) 



The same values are obtained for the area of the unitary quadrangle from eq. fll.4l) . defined 
by the unitarity relation V us V* b + V cs V* b + V ts V t * b + V t ' s V t 7 b = 0. This has to be compared with 
the amount of CPV in the three-generation SM given by llmV^V^V^V^I < 5 x 10~ 5 . 

We see that the measure of the CPV in the 4th generation model is only slightly larger 
than the amount of CPV in SM3 and this happens only for larger extra quark masses. It 
seems that extra quarks can give us new sources of large CPV phenomena, but, in general, 
cannot bring significant cumulative effect in the strength of CPV (I4.7p . Therefore, a huge 
enhancement in the Jarlskog invariant (11 .41) . in the model with the 4th family, comes 
predominantly from the large my and mf- 



5 Discussion and Conclusions 

In this paper we investigate the CP-violating decay processes involving the fourth quark 
generation and find large CP partial rate asymmetries for some of decay modes. We achieve 
that by constructing and employing global unique fit of the unitary CKM4 mass matrix. Our 
fit for certain values of the 4th generation quark mixing matrix elements for 300 < mf < 700 
GeV, produces highly enhanced acp for b' — > s decay modes. A dominance of acpib' — ► 
s {H,Z; 7,5-)) = (95,62; 47,41)% at m t > ~ 300; 350 GeV with respect to all other modes is 
particularly interesting. 

It is important to note here that all quantities appearing in the 4th generation mixing 
matrix were subject to our fitting procedure, contrary to [23 E21 EE]- So, the phases of 
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Vckm4 are fitted too, and, the complex interplay between all fitted parameters significantly 
influences the final fit of the matrix elements fl2.27ti2.3T]) . and therefore the estimated CP 
partial rate asymmetries as well. 

We have inspected FCNC decay processes of the 4th generation quarks, b' — > s X, b' — > 
bX, if — > cX, t' — > tX, with X = H, Z, 7, g, and the top decays t — > cX for 4th generation 
quarks running in the loops. The branching ratios of these rare top decays gets highly 
enhanced due to the presence of the 4th family quarks. Considering first the CPV effects 
for t — > cX modes, we have found \acp(t — > cg)\ ~ 8 — 18 % at m t i = 300 — 700 GeV; for 
t — > C7 mode acp is always bellow 6%, while for t —>■ c(H, Z) asymmetries are negligible, 
FigJHl The acp, as a function of t' mass between 300 and 700 GeV, oscillate for all decay 
modes, Figsj5J13 As already noted, the b' —>■ s (H, Z) modes with 95(62)% CP asymmetries 
at m t i = 300 GeV dominate absolutely due to the tW loop threshold at my ~ 250 GeV. 
However, acp = 47(41)% for other two modes, b' — > s(j,g), FigJT^, are more reliable as 
theoretical predictions and for measurements as well. Namely, the theoretical fact is that 
dcp{b' — * s (7, g)) receive maximal values for m t ' — 350 GeV, which is shifted away from the 
tW loop threshold. From the experimental point of view the best decay mode, out of b' — > s 
modes, is certainly b' — > s 7, because of the presence of a clean signal from the high energy 
single photon in the final state. However, the bad point is the fact that BR(b' — > sj), at 
mf = 350 GeV, could be as small as 10~ 6 , FigHK, which is at the edge of the observable region 
for the LHC. On the other hand, the shift down from m t > = 350 GeV to m t > = 300 GeV 
increases BR(b' — > 57) more than one order of magnitude (see FigHk), and only slightly 
decreases acpib' — > 57) from 47% to 42% (FigJTfci). Therefore, for my = 300,350 GeV 
the required number of b' quarks produced, in order to obtain a 3a CP violation effect, is 
Ny = 2.6 ■ 10 6 , 4.1 ■ 10 7 , respectively, which is a goal attainable after a few years of operating 
the LHC [46] at C(few 100) fb -1 . 

Comparing our estimate for acp(b' — > sj) = 47% (see Fig|7h) with very recent predictions 
of ref. [37] we have found agreement up to expected differences coming from the fitting 
procedure and the fitted CKM4 elements fl2.27ti2.31l) . 

Discussing implications for the collider experiments we conclude: There are fair chances 
for the 4th generation quarks b' and t' to be observed at LHC and that their branching ratios 
could be measured. If LHC or future colliders discover 4th generation quarks at energies 
we assumed, it is highly probable that with well executed tagging large CP partial rate 
asymmetry could be found too. 
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